Skip to content

Classify gRPC cancellation as ClientCancelled - #923

Open
KirillKurdyukov wants to merge 2 commits into
mainfrom
fix/grpc-stream-channel-close
Open

KirillKurdyukov wants to merge 2 commits into
mainfrom
fix/grpc-stream-channel-close

Conversation

@KirillKurdyukov

@KirillKurdyukov KirillKurdyukov commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Pull request type

  • Bugfix

What is the current behavior?

The SDK maps both gRPC CANCELLED and the YDB server's CANCELLED status
to ydb.Cancelled. Transport cancellation, including cancellation when a
channel closes during node restarts, is therefore not retried for idempotent
operations unless the separate retry_cancelled flag is enabled.

Async gRPC reads can also raise asyncio.CancelledError, bypassing the
normal YDB retry/reconnect classification.

What is the new behavior?

  • Introduce ydb.ClientCancelled with StatusCode.CLIENT_CANCELLED for
    gRPC cancellation. It inherits from Cancelled to preserve existing
    exception handlers.
  • Retry ClientCancelled with fast backoff only when idempotent=True,
    matching Java SDK cancellation policy. retry_cancelled=True does not
    override this requirement.
  • Keep the YDB server's CANCELLED status as Cancelled with its existing
    retry policy, and keep gRPC DEADLINE_EXCEEDED non-retriable.
  • Distinguish cancellation of an async gRPC read from caller-task cancellation.
    Preserve explicit stream cancellation and caller-task cancellation, clean up
    owned reads, and support optional disconnect callbacks.

Other information

This is the standalone SDK fix related to #921. Workflow, workload, metric and
baseline changes remain in #921.

Validation:

  • tox -e black
  • tox -e style
  • tox -e mypy
  • tox -e py -- ydb -v: 307 passed with gRPC 1.85.0.
  • sphinx-build -b html docs docs/_build/html -q: no new diagnostics.

The regression tests cover transport and YDB cancellation, deadlines,
idempotency and retry_cancelled combinations, sync and async streams,
explicit cancellation, optional callbacks, and public retry helpers.

Older local gRPC 1.78/1.82 environments exhibit an async channel-close hang,
also reproduced with the original PR iterator and test on gRPC 1.82.1.
The full unit suite retains the existing Queue.get cleanup warning.
Full YDB/chaos SLO was not run because the Docker daemon is unavailable.

Treat transport-originated stream cancellation as ConnectionLost so existing
retry and reconnect handling can recover. Preserve explicit caller and stream
cancellation, and cover Query and Topic streams with real gRPC regressions.
@codecov

codecov Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.78%. Comparing base (d6391fb) to head (31e0632).
⚠️ Report is 9 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #923      +/-   ##
==========================================
+ Coverage   82.67%   82.78%   +0.10%     
==========================================
  Files          99       99              
  Lines       12961    12985      +24     
  Branches     1269     1273       +4     
==========================================
+ Hits        10716    10749      +33     
+ Misses       1794     1789       -5     
+ Partials      451      447       -4     
Flag Coverage Δ
integration 80.43% <90.00%> (-0.02%) ⬇️
unit 50.83% <100.00%> (+1.61%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
ydb/_errors.py 94.59% <100.00%> (+3.68%) ⬆️
ydb/aio/connection.py 92.40% <100.00%> (+2.74%) ⬆️
ydb/connection.py 83.44% <100.00%> (+2.52%) ⬆️
ydb/issues.py 95.65% <100.00%> (+0.09%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@KirillKurdyukov KirillKurdyukov changed the title Fix stream recovery when gRPC channels close Classify gRPC cancellation as ClientCancelled Oct 27, 2026
@a-s-maslov

Copy link
Copy Markdown

Связанный issue: #935.

Для автоматического закрытия при влитии добавьте Fixes #935 в описание PR.

@KirillKurdyukov
KirillKurdyukov marked this pull request as ready for review November 3, 2026 06:55
@vgvoleg
vgvoleg requested a lite review from Copilot November 6, 2026 09:08

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Approval is blocked by an unresolved moderate connection-eviction bug on synchronous transport cancellation, with an additional tracing-status mapping omission.

2 open issues
What changed in this PR

This pull request distinguishes transport-level gRPC cancellation from YDB server-side cancellation, enabling retries for idempotent operations while preserving explicit caller cancellation and deadline behavior across synchronous and asynchronous APIs.

Changes:

  • Added ClientCancelled and StatusCode.CLIENT_CANCELLED.
  • Updated cancellation classification, retry behavior, stream cleanup, and disconnect callbacks.
  • Added regression coverage and documented the new exception and retry semantics.
File Description
ydb/​issues.py Defines the new client-cancellation status and exception.
ydb/​connection.py Maps synchronous gRPC cancellation and updates stream cancellation handling.
ydb/​aio/​connection.py Separates async gRPC read cancellation from caller-task cancellation.
ydb/​_utilities_test.py Adds cancellation, deadline, retry, callback, and stream regression tests.
ydb/​_errors.py Implements idempotency-based retry behavior for ClientCancelled.
docs/​errors.rst Documents the exception hierarchy and retry policies.
docs/​apireference.rst Exposes ClientCancelled in the API reference.
CHANGELOG.md Records the user-visible cancellation behavior change.

Review effort: Lite

Comment thread ydb/connection.py
return issues.Unimplemented("Method or feature is not implemented on server!")
elif rpc_error.code() == grpc.StatusCode.CANCELLED:
return issues.Cancelled(rpc_error.details())
return issues.ClientCancelled(rpc_error.details())
Comment thread ydb/issues.py
DEADLINE_EXCEEDED = _TRANSPORT_STATUSES_FIRST + 30
CLIENT_INTERNAL_ERROR = _TRANSPORT_STATUSES_FIRST + 40
UNIMPLEMENTED = _TRANSPORT_STATUSES_FIRST + 50
CLIENT_CANCELLED = _TRANSPORT_STATUSES_FIRST + 60

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants